Virtual reality head-mounted devices having reduced numbers of cameras, and methods of operating the same

ABSTRACT

Example virtual-reality head-mounted devices having reduced numbers of cameras, and methods of operating the same are disclosed herein. A disclosed example method includes providing a virtual-reality (VR) head-mounted display (V-HMD) having an imaging sensor, the imaging sensor including color-sensing pixels, and infrared (IR) sensing pixels amongst the color-sensing pixels; capturing, using the imaging sensor, an image having a color portion and an IR portion; forming an IR image from at least some of the IR portion from the image; performing a first tracking based on the IR image; forming a color image by replacing the at least some of the removed IR portion with color data determined from the color portion of the image and the location of the removed IR-sensing pixels in the image; and performing a second tracking based on the color image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of, and claims priority to, U.S.patent application Ser. No. 14/996,858, filed on Jan. 15, 2016, titled“VIRTUAL REALITY HEAD-MOUNTED DEVICES HAVING REDUCED NUMBERS OF CAMERAS,AND METHODS OF OPERATING THE SAME,” now, U.S. Pat. No. 10,114,465, thedisclosure of which is incorporated herein in its entirety.

FIELD OF THE DISCLOSURE

This disclosure relates generally to virtual reality, and, moreparticularly, to virtual-reality head-mounted devices having reducednumbers of cameras, and methods of operating the same.

BACKGROUND

Virtual-reality head-mounted displays have multiple cameras to imageand/or render virtual-reality environments in which someone can bephysically or virtually present, and to track movements of the viewerand/or other items physically and/or virtually present in thevirtual-reality environment.

SUMMARY

Virtual-reality head-mounted devices or displays having reduced numbersof cameras, and methods of operating the same are disclosed. Andisclosed example method includes providing a virtual-realityhead-mounted display having an imaging sensor, the imaging sensorincluding color-sensing pixels, and infrared sensing pixels amongst thecolor-sensing pixels; capturing, using the imaging sensor, an imagehaving a color portion and an infrared portion; forming an infraredimage from at least some of the infrared portion from the image;performing a first tracking based on the infrared image; forming a colorimage by replacing the at least some of the removed infrared portionwith color data determined from the color portion of the image and thelocation of the removed infrared-sensing pixels in the image; andperforming a second tracking based on the color image.

A disclosed example virtual-reality head-mounted device for use in avirtual-reality environment includes an imaging sensor to capture animage using color-sensing pixels, and infrared sensing pixels locatedamongst the color-sensing pixels, the captured image having a colorportion and an infrared portion; a reconstructor configured to remove atleast some of infrared portion from the image, and form an infraredimage from the removed infrared portion; a first tracker configured toperform first virtual-reality tracking within the virtual-realityenvironment using the infrared image; an image modifier to form a colorimage by substituting the removed infrared portion with color-sensingpixels determined from the color portion of the image and the locationsof the removed infrared-sensing pixels in the image; and a secondtracker configured to carry out a second virtual-reality tracking basedon the color image.

A disclosed example non-transitory machine-readable media storesmachine-readable instructions that, when executed, cause a machine to atleast provide a virtual-reality head-mounted display having an imagingsensor, the imaging sensor including color-sensing pixels, and infraredsensing pixels amongst the color-sensing pixels; capture, using theimaging sensor, an image having a color portion and an infrared portion;form an infrared image using at least some of the infrared portion;perform a first tracking based on the infrared image; form a color imageby replacing the at least some of the infrared portion of the with colordata determined from the color portion of the image and the location ofthe at least some of the infrared portion in the image; and perform asecond tracking based on the color image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of an example virtual-reality systemincluding a head-mounted device having fewer cameras in accordance withthe teachings of this disclosure.

FIG. 2 is an example front view of the disclosed example virtual-realityhead-mounted display of FIG. 1 in accordance with the teachings of thisdisclosure.

FIG. 3 is a schematic diagram of an example virtual-reality head-mounteddisplay having fewer cameras in accordance with the teachings of thisdisclosure.

FIG. 4 is an example imaging sensor in accordance with the teachings ofthis disclosure.

FIG. 5 is a diagram illustrating an example extraction of infraredsensing pixels.

FIGS. 6A and 6B are a flowchart illustrating an example method that may,for example, be implemented using machine-readable instructions executedby one or more processors to operate the example head-mounted displaysdisclosed herein.

FIG. 7 is a block schematic diagram of an example computer device and anexample mobile computer device that may be used to implement theexamples disclosed herein.

DETAILED DESCRIPTION

Virtual-reality (VR) head-mounted displays or devices (V-HMDs) arenext-generation computing platforms for providing virtual realitysystems and/or environments. A V-HMD can include multiple sub-systems,such as a display sub-system, a camera sub-system, an image processingsub-system, a controller sub-system, etc. There is need for camera andimage processing sub-systems that can perform, among other things, aplurality of VR functions that can meet customer, user, and/or wearerexpectations for VR functionality. Example VR functions include, but arenot limited to, 6 degree-of-freedom (6DoF) head tracking, finger/handtracking, environmental (or depth) sensing, pass-through of images fordisplay via or in the V-HMD, tracking a VR controller or other deviceheld by a user, etc. Current V-HMDs designs require dedicated camera(s)for each VR function being implemented. For example, 1^(st) and 2^(nd)cameras for head tracking, 3^(rd) and 4^(th) cameras for finger/handtracking, 5^(th) camera for environmental sensing (e.g., depth), 6^(th)and 7^(th) cameras for pass-through, and 8^(th) and 9^(th) cameras fortracking a VR controller, etc. Thus, current V-HMDs can require nine ormore cameras to provide this basic set of VR functions. The fact thatconventional V-HMDs need so many cameras, presents numerous andsignificant disadvantages to overcome, especially given a large numberof V-HMDs may be retail devices where looks, size, industrial design,weight, etc. are important. For example, weight increase, size increase,cost increase, number of components increase, decreased reliability,increased complexity, industrial design limitations, etc. These camerastypically differ from other cameras of a V-HMD in, for example, size,angle of view, resolution, etc. Future needs for additional VR functionsfurther compounds these issues. Limiting the number of camerascorrespondingly reduces the number of VR functions that can be realized,thus, making such a VR-HMD less attractive in the marketplace. All said,there are significant hurdles in conventional V-HMDs that must beovercome to meet the needs in the market, which cannot be met byconventional V-HMDs. The fact that V-HMDs being able to support such alarge number of VR functions is a testament to a significant unmet need.

Example V-HMDs that overcome at least these problems are disclosedherein. The disclosed examples require fewer cameras, two (2) versusnine (9), while still providing at least the same set of VR functionsdelineated above. The cameras, and the imaging sensors therein, aredesigned to perform a wide-range of imaging capabilities using fewercameras. An example camera can be designed to, for example, perform aset of camera capabilities consisting of, for example, a widefield-of-view, a maximum and/or minimum sampling rate, a needed pixelarrangement, etc. to realize the desired set of VR functions to berealized. For instance, an example output of an example wide-anglecapable imaging sensor as disclosed herein can be cropped to form amoderate, a normal, a narrow, etc. field-of-view image. Thus, a wideangle camera in accordance with the teachings of this disclosure can beused to capture data needed to simultaneously and/or sequentiallyrealize a narrow, a normal and/or a wide field-of-view. Because thesuper-set of capabilities addresses or provides each VR functions, cost,complexity, weight, etc. of using the conventional V-HMD to implements alarge number of VR functions. Realizing support for a larger number ofVR functions using only a pair of stereoscopic cameras provides asignificant new advantage that can be brought to bear in the V-HMDmarketplace. For example, an imaging sensor and its associated cameracould be designed to support a wide angle of view that can support theneeded angle-of-view, and be processed to extract (cropped) to get imagedata for a normal angle-of-view. That is, the camera(s) can be viewed asbeing able to support the requirements of the two or more VR functions.In general, a camera or its imaging sensor has be designed such that allof the data needed for all supported VR function can be obtained byprocessing the image data in a respective or corresponding manner. Asdisclosed herein, a plurality of VR functions can be realized using onlya pair of cameras, which is about a 75% decrease in the required numberof cameras. It is contemplated that other past, present or futurefunctions may also be supported by the two camera configurationdisclosed herein.

Reference will now be made in detail to non-limiting examples of thisdisclosure, examples of which are illustrated in the accompanyingdrawings. The examples are described below by referring to the drawings,wherein like reference numerals refer to like elements. When likereference numerals are shown, corresponding description(s) are notrepeated and the interested reader is referred to the previouslydiscussed figure(s) for a description of the like element(s). Theseexamples and variants and portions thereof shown in the attacheddrawings are not drawn to scale, with specific shapes, or with specificrelative dimensions as they are not important to this disclosure and mayrender the drawings more difficult to comprehend. Specific elements mayhave been intentionally exaggerated for discussion purposes. Instead,the drawings have been drawn for clarity and comprehension. Further, thearrangement of elements and couplings maybe changed, rearranged, etc.according to other implementations of this disclosure and the claimsherein.

Turning to FIG. 1, a block diagram of an example VR system 100 is shown.The example VR system 100 includes a V-HMD 110 having a front face 112in accordance with the teachings of this disclosure are shown. As shownin FIGS. 2 and 3, only a pair of cameras 113 and 114 is needed at thefront face 112. The cameras 113 and 114 can be matched and separated byabout the same distance as a person's eyes so that, if a pair ofpictures is taken the pictures can provide collectively a stereoscopicimpression. For example, in FIG. 2, the cameras 113 and 114 face upwardfrom the page, and face leftward in the example of FIG. 3. The cameras113 and 114 can capture a portion of the light moving toward orimpinging the face 112. The light can move toward the face 112 alongdifferent paths or trajectories. The position(s) of the cameras 113 and114 may vary based on any number and/or type(s) of design parameters.For example, the space between the cameras 113 and 114 can impactperceived depth of field. They may also be selected based on a desiredV-HMD 110 size, industrial design, use of additional cameras,anticipated size of wearer, additional VR functions or featuresincluded, etc. Additional and/or alternative cameras can be includedand/or used as an upgrade, to e.g., support additional VR functions thatwe not previously contemplated. For example, a fish eye lense could nothave been originally included, but could later be installed to supportnewly contemplated VR functions. Further, one or more of the cameras 113and 114 may be selectively controllable to image different locations atdifferent times. Further still, the cameras 113 and 114 need not be thesame. Moreover, a camera 113, 114 may be, e.g., updated by a user orservice center to support additional V-HMD functions, and/or to modifyor customize functionality of a V-HMD. As will be discussed morethoroughly below, a user 135 can have a VR controller 136. The VRcontroller 136 can, e.g., emit and/or reflect infrared (IR) or any othertype(s) of light that can be detected by one or more of the cameras 113and 114 to help determine positions of, for example, the user's hands.Likewise, other elements 115 in the VR system 100 can emit and/orreflect IR or other type(s) of light for tracking or other VR purposes.

In general, the example VR system 100 provides a VR environment and VRcontent that can be accessed, viewed, and/or otherwise interacted with.As will be described below in connection with FIG. 3, using only two ofthe disclosed example cameras 113 and 114, the example V-HMD 110 canfacilitate the implementation of multiple VR functions rather thanhaving to implement an impractical larger number of cameras, as isrequired in conventional V-HMDs (e.g., see a discussed above)

As shown in FIG. 1, the example VR system 100 includes a plurality ofcomputing and/or electronic devices that can exchange data over anetwork 120. The devices may represent clients or servers, and cancommunicate via the network 120 or any other additional and/oralternative network(s). Example client devices include, but are notlimited to, a mobile device 131 (e.g., a smartphone, a personal digitalassistant, a portable media player, etc.), an electronic tablet, alaptop or netbook 132, a camera, the V-HMD 110, a desktop computer 133,a gaming device, and any other electronic or computing devices that cancommunicate using the network 120 or other network(s) with othercomputing or electronic devices or systems, or that may be used toaccess VR content or operate within a VR environment. The devices 110and 131-133 may represent client devices. In some examples, the devices110 and 131-133 include one or more processors and one or more memorydevices, which can execute a client operating system and one or moreclient applications that can access, control, and light-emitting portionVR content on a light-emitting portion device implemented together witheach respective device. One or more of the devices 110 and 131-133 can,e.g., emit or reflect infrared (IR) or other type(s) of light that canbe detected by one or more of the cameras 113 and 114 to help determineposition of a user or the devices 110, 131-133 for tracking or other VRfunctions.

An example stereoscopic placement of the cameras 113 and 114 is shown inFIG. 2. In the example of FIG. 2, the cameras 113 and 114 are equallyspaced from opposite sides of a virtual dividing line 205, and arepositioned the same different from the bottom 210 of the face 112. Othercamera configurations reflecting other and/or alternative implementationobjectives and/or desired VR functions are contemplated.

Turning now to FIG. 3, a schematic diagram of an example imagingpipeline 300 that may be used with any of the example V-HMDs disclosedherein shown. As discussed below, the imaging pipeline 300 can be usedto carry out other non-imaging function(s) 335. The imaging pipeline 300includes a camera portion 305 that includes the cameras 113 and 114, animage processing portion 310 that forms the necessary images for VRtracking functions, and possibly in some instances, other portion(s) 335that can perform other imaging function(s) and/or non-imagingfunction(s). FIG. 3 shows the logical arrangements of elements, blocks,functions, etc. For clarity of illustration and ease of comprehension,details such as busses, memory, caches, etc. are omitted, but inclusionand implementation of such details are well known to those of skill inthe art.

The example camera portion 305 of FIG. 1 includes the cameras 113 and114 together with respective lenses 306 and 307. In the orientation ofFIG. 3, light 308A and 308B moving from left to right, passes throughthe lenses 306, 307, possibly with optical effects performed by thelenses 306, 307, and impinges on the RGIBIR imaging sensors of thecameras 113, 114. Lenses such as 306 and 307 may be fixed lenses, or canhave selectively variable focal lengths, selectively perform focusing,have selectively variable apertures, have selectively variable depth offield, etc. These selective functions can be perform automatically,manually, or some combination thereof.

The two cameras 113 and 114 can be identical to each other, with bothhaving red (R) sensing pixels (one of which is designated at referencenumeral 405), green (G) sensing pixels (one of which is designated at406 reference numeral 406), blue (B) sensing pixels (one of which isdesignated at reference numeral 407), and infrared (IR) sensing pixels(one of which is designated at reference numeral 408). The pixels405-408 can be arranged in a regular or semi-random pattern, such as theexample pattern shown in FIG. 4. In the example of FIG. 4, some of the Gsensing 406 pixels of a conventional Bayer RGB sensor are replaced withIR sensing pixels 408 in accordance with the teachings of thisdisclosure. Accordingly, the IR sensing pixels 408 are placed, located,etc. within, amongst, etc. the color sensing R, G and B pixels 405-407.Compared with conventional Bayer RGB sensors that can be used in digitalcameras, the RGBIR sensor 400 disclosed herein can have a mixture of R,G and B pixels sensitive to red, green and blue visible light, and IRpixels sensitive to non-visible infrared light (e.g., see FIG. 4).Accordingly, the imaging sensors of the cameras 113, 114 are RGBIRsensors, and images output by the cameras are RGBIR images. That is, theRGBIR images output by the cameras 113, 114 convey red, green, blue andIR information. As shown, a 2×2 block of pixels can include 1 red pixel,1 green pixel, 1 blue pixel and 1 IR pixel. Color and IR pixels may bearranged in other pattern(s), and/or with different ratio(s).

The R, G and B pixels 405-407 can be equipped with a per-pixel IR-cutfilter so a pixel is not sensitive or responsive to IR light.Additionally or alternatively, a dual-band filter that passes onlyvisible light and a narrow band of IR light can be placed inside thecamera module such that the R, G and B pixels 405-407 will only beresponse to light, i.e., an IR-cut filter per pixel. With thisarrangement, IR sensing pixels only sense narrowband IR light, and theR, G and B pixels 405-407 only sense R, G and B visible light. This waythe color image produced by the R, G and B sensing pixels 405-407 wouldbe improved compared to no per-pixel IR cut filter on top of the R, Gand B sensing pixels 405-407 because no IR light is leaked into those R,G and B sensing pixels 405-407.

In some instances, the camera portion 305 includes an IR emitter 309.The example IR emitter 309 can be selectively operated or activated toemit IR light that can reflect off objects such as the example objects115 and 136 (see FIG. 1), allowing the reflected IR light to be used tolocate the objects 115, 136. Additionally or alternately, the IR emitter309 may be implemented separately from the V-HMD 110.

Turning to the example image processing portion 310 of FIG. 3, RGBIRimages 313A, 314A captured by respective ones of the cameras 113, 114are provided to a reconstructor 311. Using any number and/or type(s) ofmethod(s), techniques(s), algorithm(s), circuit(s), etc., thereconstructor 311 can create an RBG image 313B, 314B and an IR image313C, 314C for respective ones of RGBIR images 313A, 314A provided bythe cameras 113, 114.

As shown in FIG. 5, the IR images 313C and 314C in FIG. 3 can be formedby collecting the values of the IR pixels from a full RGBIR image array500 into a smaller array 550 of just IR pixel data. Illustrated in FIG.5 are three example movements of IR pixels 511, 517 and 519 from theimage 500 into the image 550. The other IR pixels can be moved orcollected in a similar manner.

The RGB images 313B and 314B may be formed by extracting or removing theIR pixels from an RGBIR image. As they are removed or extracted, thevalues in the array where the IR pixels were removed represent can begiven a NULL, vacant, etc. value or indicator. In some examples, the IRpixels need not be so modified as they can be later overwritten bysubsequent image processing. In the example of FIG. 4, the IR pixelsreplaced every other green pixels, however, other patterns could beused.

Using any number and/or type(s) of method(s), techniques(s),algorithm(s), circuit(s), etc., the reconstructor 311 can create an RBGimage 313B, 314B and an IR image 313C, 314C for respective ones of RGBIRimages 313A, 314A provided by the cameras 113, 114. Using any numberand/or type(s) of method(s), techniques(s), algorithm(s), circuit(s),etc., the reconstructor 311 can create fills in the blank or vacant IRpixel locations with suitable green pixel values, thus, forming acompleted RGB image 313B, 314B.

Turning now to how the four images 313B, 313C, 314B and 314C in FIG. 3are created, formed or generation by the reconstructor 311 can beadaptive, changed, combined, etc. to perform a number of V-HMD functionsand/or, more broadly, VR functions. Because of the various processing ofthe images 313A and 314A, the reconstructor 311 makes the images 313B,313C, 314B and 314C that give the impression that more than two cameraswere used to capture the images 313B, 313C, 314B and 314C.

The two RGB images 313B and 314B can be processed by a Chromasub-sampler 315.

To perform 6DoF tracking, the example imaging pipeline 300 includes anynumber and/or type(s) of a 6DoF processor(s) 316. Using any numberand/or type(s) of method(s), techniques(s), algorithm(s), circuit(s),etc., the 6DoF processor 316 can process the Chroma sub-sampled imagesprovided by the Chroma sub-sampler 315 to track stationary and/or movingscene features in these the images provided by the Chroma sub-sampler315. In some examples, only luminance data is used, possibly withinertial measurement data (IMD) 317 provided by an inertial measurementunit (IMU) 318. The 6DoF processor 316 provides determined 6DoF trackingdata 332 the VR function portion 330 for further processing.

The example image processing portion 310 can also include a stereoscopicimage former 319 and a cropper/scaler 320 to support tracking of, forexample, hands and fingers. Using any number and/or type of method(s),techniques(s), algorithm(s), circuit(s), etc., the stereoscopic imageformer 319 combines the RGB images 313B and 314B, and the IR images 313Cand 314C to form one or more stereoscopic images (e.g.,three-dimensional (3D) data) of, among other things, the position andmovement fingers and/or hands. This data can be provided to afinger/hand tracker 334 in the VR tracking portion 330. The IR images313C and 314C can be formed using IR emitters and/or reflectors on theuser's hand or elsewhere in a VR environment or system. Because thecamera 113 and 114 can have a wider field of view (FOV) than the fieldof view needed to perform finger and/or hand tracking, thecropper/scaler 320 can crop the image to a smaller size and, whenappropriate, can sub-sample the stereo images before tracking isperformed. Such a reduction can be useful in reducing processing powerand/or memory requirements.

Furthermore, the RGB images 313B and 314B can be used as pass-throughimages. In other words, the images 313B and 314B can be passed to thedisplaying module 336 of a V-HMD. The images 313B and 314B can becropped/scaled by a cropper/scaler 322 to fit to the display resolutionand/or field-of-view of the V-HMD, enabling a user to see, move and/orotherwise track themselves in a mixed virtual and real environment, andenabling augmented reality tracking in which virtual objects are laid ontop of a real world scene. The pass-through images may be used for anynumber and/or type(s) of other functions.

The two IR images 313C and 314C can be used by an optical tracker 338 todetect IR light emitted or reflected by the VR controller such as thecontroller 136 so as to optically track its location. Accordingly, theIR images 313C and 314C can be passed to an optical tracker of the VRfunction portion 330 as a QHD IR pair stereo vision can be used toobtain depth information. To increase the robustness of stereo vision,one can add a patterned illuminator to the system such that structuredlight would be projected to the scene to aid stereo vision underlow-light situation. This alternative implementation uses an additionalcomponent then disclosed above. Thus, there may be a tradeoff betweenlow-light depth performance versus complexity and/or cost.

In some examples, a portion of the imager processing portion 310 of FIG.3 related to signal processing is implemented using, for example, aprogrammable processor, such as the Movidius Myriad 2 Vision ProcessingUnit (VPU). The VR function portion 330 may be implemented using, forexample, an all-in-one mobile processor, such as the Qualcomm©Snapdragon™ processor. The image processing pipeline can, additionallyor alternatively, be implemented by one or more Atmel®, Intel®, AMD®,and/or ARM® microprocessors. Of course, other (micro-) processors fromother processor families and/or manufacturers are also appropriate.

One or more of the elements and interfaces shown in FIGS. 1-3 may becombined, divided, re-arranged, omitted, eliminated and/or implementedin any other way. Further, one or more circuit(s), programmableprocessor(s), fuses, application-specific integrated circuit(s)(ASIC(s)), programmable logic device(s) (PLD(s)), field-programmablelogic device(s) (FPLD(s)), and/or field-programmable gate array(s)(FPGA(s)), etc. can be used. Moreover, more interfaces and/or elementsmay be included instead of, or in addition to, those shown, and/or mayinclude more than one of any or all of the illustrated interfaces andelements. The elements shown may, for example, be implemented asmachine-readable instructions carried out by one or more processors. Aprocessor, a controller and/or any other suitable processing device suchas those shown in FIG. 7 may be used, configured and/or programmed toexecute and/or carry out the examples disclosed herein. For example, thedisclosed examples may be embodied in program code and/ormachine-readable instructions stored on a tangible and/or non-transitorycomputer-readable medium accessible by a processor, a computer and/orother machine having a processor, such as that discussed below inconnection with FIG. 7. Machine-readable instructions comprise, forexample, instructions that cause a processor, a computer and/or amachine having a processor to perform one or more particular processes.Many other methods of implementing the disclosed examples may beemployed. For example, the order of execution may be changed, and/or oneor more of the blocks and/or interactions described may be changed,eliminated, sub-divided, or combined. Additionally, any or the entireexample may be carried out sequentially and/or carried out in parallelby, for example, separate processing threads, processors, devices,discrete logic, circuits, etc.

Turning collectively to FIGS. 6A and 6B, an example method 600 that maybe performed to control and operate the example V-HMDs disclosed hereinis shown.

The example method of FIGS. 6A and 6B begins with providing a V-HMDhaving first and second stereoscopically arranged RGBIR cameras (block602), and capturing first and second RGBIR images using respective onesof the first and second RGBIR cameras (block 604). First and second IRimages are formed for respective ones of the first and second RGBIRimages by extracting IR pixels from respective ones of the capturedfirst and second RGBIR images (block 606 and 608). First and second RGBimages are formed by, for example, interpolating the color data to fillin the IR locations in respective ones of the RGBIR images that wereextracted, removed, etc. (block 610 and 612).

If optical tracking is to be performed (block 614), the IR portion ofthe original first and second RGBIR images 313A and 314A can beextracted from the first and second RGBIR images, and the IR trackingdata provided to the optical tracker 338 (block 616). The QHD IR pairstereo vision can be used to obtain depth information. To increase therobustness of stereo vision, a patterned illuminator can be added to thesystem such that structured light would be projected to the scene to aidstereo vision under low-light situation. This alternative implementationuses an additional component then disclosed above. Thus, there may be atradeoff between low-light depth performance versus complexity and/orcost

If pass through images are to be provided (block 618), the first andsecond RGB images 313B and 314B can be used as first and secondpass-through images (block 620) The first and second images 313B and314B can be cropped/scaled by the cropper/scaler 322 to fit to thedisplay resolution and/or field-of-view of the V-HMD, enabling thewearer see the real world and enabling augmented reality in whichvirtual objects are laid on top of a real world scene. The pass-throughimages may be used for any number and/or type(s) of other function.

Continuing at block 622 of FIG. 6B, if IR finger/hand tracking is to beperformed (block 622), the first and second IR portions of the originalfirst and second RGBIR images 313A and 314A, and the color portions ofthe first and second RGBIR images, with the first and second extractedIR portions replaced with color data can be provided to the finger/handtracker 334 (block 624). The finger/hand tracker 334 can process theimage data to track finger/hand position and/or movement (block 626).

If 6DoF tracking is to be performed (block 628), the 6DoF processor(s)316 can process Chroma sub-sampled images provided by the Chromasub-sampler 315 to track stationary and/or moving scene features inthese the first and second images provided by the Chroma sub-sampler 315(block 630). In some examples, only luminance data is used, possiblywith inertial measurement data (IMD) 317 provided by an inertialmeasurement unit (IMU) 318. The 6DoF processor 316 provides determined6DoF tracking data 332 the VR function portion 330 for furtherprocessing. Control then exits from the example method of FIGS. 6A and6B.

The example methods of FIGS. 6A and 6B, or other methods disclosedherein, may, for example, be implemented as machine-readableinstructions carried out by one or more processors to control or operatethe example display assemblies disclosed herein. A processor, acontroller and/or any other suitable processing device may be used,configured and/or programmed to execute and/or carry out the examplemethods disclosed herein. For instance, the example methods of FIGS. 6Aand 6B may be embodied in program code and/or machine-readableinstructions stored on a tangible and/or non-transitorycomputer-readable medium accessible by a processor, a computer and/orother machine having a processor, such as that discussed below inconnection with FIG. 7. Machine-readable instructions comprise, forexample, instructions that cause a processor, a computer and/or amachine having a processor to perform one or more particular processes.Many other methods of implementing the example methods of FIGS. 6A and6B may be employed. For example, the order of execution may be changed,and/or one or more of the blocks and/or interactions described may bechanged, eliminated, sub-divided, or combined. Additionally, any of theentire example methods of FIGS. 6A and 6B may be carried outsequentially and/or carried out in parallel by, for example, separateprocessing threads, processors, devices, discrete logic, circuits, etc.

As used herein, the term “computer-readable medium” is expressly definedto include any type of tangible or non-transitory computer-readablemedium and to expressly exclude propagating signals. Examplecomputer-readable medium include, but are not limited to, a volatileand/or non-volatile memory, a volatile and/or non-volatile memorydevice, a compact disc (CD), a digital versatile disc (DVD), a read-onlymemory (ROM), a random-access memory (RAM), a programmable ROM (PROM),an electronically-programmable ROM (EPROM), an electronically-erasablePROM (EEPROM), an optical storage disk, an optical storage device, amagnetic storage disk, a magnetic storage device, a cache, and/or anyother storage media in which information is stored for any duration(e.g., for extended time periods, permanently, brief instances, fortemporarily buffering, and/or for caching of the information) and thatcan be accessed by a processor, a computer and/or other machine having aprocessor.

Returning to FIG. 1, the example network 120 may be constructed usingany number and type(s) of private and/or public networks including, butnot limited to, the Internet, a cellular data network, a coaxial cablenetwork, a satellite network, a fiber optic network, a dialup orbroadband modem over a telephone network, a Wi-Fi® hotspot, a privatecommunications network (e.g., a private local area network (LAN), awireless local area network (WLAN), a leased line), etc., and anycombination thereof.

The example system 100 of FIG. 1 further includes a VR content system140. The VR content system 140 may represent a server device. Theexample VR content system 140 of FIG. 1 includes any number ofrepositories 142 storing content and/or virtual reality applications 144that can generate, modify, and execute VR scenes.

The example head-mounted display 110 of FIG. 1 may include, forinstance, a VR headset, glasses, an eyepiece, or any other wearabledevice capable of light-emitting portioning VR content. In operation,the head-mounted display 110 can, for example, execute a VR application144 to playback, present or light-emitting portion received or processedimages for a user. However, images maybe played back, presented andlight-emitting portioned by the head-mounted display 110 without needfor a VR application 144. In some implementations, a VR application 144of the head-mounted display 110 is hosted by one or more of the devices131-133 shown in FIG. 1.

The one or more VR applications 144 of FIG. 1 can be configured toexecute on any or all of the devices 110 and 131-133. The head-mounteddisplay 110 can be communicatively coupled to one or more of the devices110 and 131-133 to access VR content stored on or accessible via the VRcontent system 140. The devices 131-133 can be communicatively coupled(wired and/or wirelessly) to the head-mounted display 110, which canprovide VR content for light-emitting portion on the head-mounteddisplay 110.

The example head-mounted display 110 may be wirelessly coupled to thedevices 131-133 via any combination of wireless networks and/orprotocols such as, but not limited to, any of the Institute ofElectrical and Electronics Engineers (IEEE®) 802.11x family ofstandards, Wi-Fi®, Bluetooth®, etc.

In the event the head-mounted display 110 is electrically coupled to oneor more of the devices 131-133, a cable with an appropriate connector oneither end for plugging into the devices 110 and 131-133 may be used.For example, the cable can include a Universal Serial Bus (USB)connector on both ends. The USB connectors can be the same USB typeconnector, or the USB connectors can each be a different type of USBconnector. The various types of USB connectors include, but are notlimited to, USB A-type connectors, USB B-type connectors, micro-USB Aconnectors, micro-USB B connectors, micro-USB AB connectors, USB fivepin Mini-b connectors, USB four pin Mini-b connectors, USB 3.0 A-typeconnectors, USB 3.0 B-type connectors, USB 3.0 Micro B connectors, andUSB C-type connectors.

In some implementations, the mobile device 131 executes the VRapplication(s) 144 and provides the content for the VR environment. Insome implementations, the laptop computing device 132 executes the VRapplication(s) 144 and provides content from one or more content servers(e.g., the VR content server 140). In some implementations, the desktopcomputing device 133 executes the VR application(s) 144 and providescontent from one or more content servers (e.g., the VR content server140). The one or more content servers 140 and one or morecomputer-readable storage devices 142 can communicate with the mobiledevice 131, the laptop computing device 132, and/or the desktopcomputing device 133 using the network 120 to provide content forlight-emitting portion in the head-mounted display 110.

Returning to FIG. 1, the example network 120 may be constructed usingany number and type(s) of private and/or public networks including, butnot limited to, the Internet, a cellular data network, a coaxial cablenetwork, a dialup or broadband modem over a telephone network, a Wi-Fi®hotspot, a private communications network (e.g., a private local areanetwork (LAN), a wireless local area network (WLAN), a leased line),etc.

The example system 100 of FIG. 1 further includes a VR content system140. The VR content system 140 may represent a server device. Theexample VR content system 140 of FIG. 1 includes any number ofrepositories 142 storing content and/or virtual reality applications 144that can generate, modify, and execute VR scenes.

The example V-HMD 110 of FIG. 1 may include, for instance, a V-HMD,glasses, an eyepiece, or any other wearable device capable oflight-emitting portioning VR content. In operation, the V-HMD 110 can,for example, execute a VR application 144 that can playback, present orlight-emitting portion received or processed images for a user. However,images maybe played back, presented and light-emitting portioned by theV-HMD 110 without need for a VR application 144. In someimplementations, a VR application 144 of the V-HMD 110 is hosted by oneor more of the devices 131-133 shown in FIG. 1.

The one or more VR applications 144 of FIG. 1 can be configured toexecute on any or all of the devices 110 and 131-133. The V-HMD 110 canbe communicatively coupled to one or more of the devices 110 and 131-133to access VR content stored on or accessible via the VR content system140. The devices 131-133 can be communicatively coupled (wired and/orwirelessly) to the V-HMD 110, which can provide VR content forlight-emitting portion on the V-HMD 110.

In the event the V-HMD 110 is electrically coupled to one or more of thedevices 131-133, a cable with an appropriate connector on either end forplugging into the devices 110 and 131-133 may be used. For example, thecable can include a Universal Serial Bus (USB) connector on both ends.The USB connectors can be the same USB type connector, or the USBconnectors can each be a different type of USB connector. The varioustypes of USB connectors include, but are not limited to, USB A-typeconnectors, USB B-type connectors, micro-USB A connectors, micro-USB Bconnectors, micro-USB AB connectors, USB five pin Mini-b connectors, USBfour pin Mini-b connectors, USB 3.0 A-type connectors, USB 3.0 B-typeconnectors, USB 3.0 Micro B connectors, and USB C-type connectors.

In some implementations, the mobile device 131 executes the VRapplication(s) 144 and provides the content for the VR environment. Insome implementations, the laptop computing device 132 executes the VRapplication(s) 144 and provides content from one or more content servers(e.g., the VR content server 140). In some implementations, the desktopcomputing device 133 executes the VR application(s) 144 and providescontent from one or more content servers (e.g., the VR content server140). The one or more content servers 140 and one or morecomputer-readable storage devices 142 can communicate with the mobiledevice 131, the laptop computing device 132, and/or the desktopcomputing device 133 using the network 120 to provide content forlight-emitting portion in the V-HMD 110.

Turning to FIG. 7, an example of a generic computer device P00 and ageneric mobile computer device P50, which may be used with thetechniques described here. Computing device P00 is intended to representvarious forms of digital computers, such as laptops, desktops, tablets,workstations, personal digital assistants, televisions, servers, bladeservers, mainframes, and other appropriate computing devices. Computingdevice P50 is intended to represent various forms of mobile devices,such as personal digital assistants, cellular telephones, smart phones,and other similar computing devices. The components shown here, theirconnections and relationships, and their functions, are meant to beexemplary only, and are not meant to limit implementations of theinventions described and/or claimed in this document.

Computing device P00 includes a processor P02, memory P04, a storagedevice P06, a high-speed interface P08 connecting to memory P04 andhigh-speed expansion ports P10, and a low speed interface P12 connectingto low speed bus P14 and storage device P06. The processor P02 can be asemiconductor-based processor. The memory P04 can be asemiconductor-based memory. Each of the components P02, P04, P06, P08,P10, and P12, are interconnected using various busses, connections,memories, caches, etc. and may be mounted on a common motherboard or inother manners as appropriate. The processor P02 can process instructionsfor execution within the computing device P00, including instructionsstored in the memory P04 or on the storage device P06 to light-emittingportion graphical information for a GUI on an external input/outputdevice, such as light-emitting portion P16 coupled to high speedinterface P08. In other implementations, multiple processors and/ormultiple buses may be used, as appropriate, along with multiple memoriesand types of memory. Also, multiple computing devices P00 may beconnected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory P04 stores information within the computing device P00. Inone implementation, the memory P04 is a volatile memory unit or units.In another implementation, the memory P04 is a non-volatile memory unitor units. The memory P04 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device P06 is capable of providing mass storage for thecomputing device P00. In one implementation, the storage device P06 maybe or contain a computer-readable medium, such as a floppy disk device,a hard disk device, an optical disk device, or a tape device, a flashmemory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product can be tangibly embodied inan information carrier. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The information carrier is a computer- ormachine-readable medium, such as the memory P04, the storage device P06,or memory on processor P02.

The high speed controller P08 manages bandwidth-intensive operations forthe computing device P00, while the low speed controller P12 manageslower bandwidth-intensive operations. Such allocation of functions isexemplary only. In one implementation, the high-speed controller P08 iscoupled to memory P04, light-emitting portion P16 (e.g., through agraphics processor or accelerator), and to high-speed expansion portsP10, which may accept various expansion cards (not shown). In theimplementation, low-speed controller P12 is coupled to storage deviceP06 and low-speed expansion port P14. The low-speed expansion port,which may include various communication ports (e.g., USB, Bluetooth,Ethernet, wireless Ethernet) may be coupled to one or more input/outputdevices, such as a keyboard, a pointing device, a scanner, or anetworking device such as a switch or router, e.g., through a networkadapter.

The computing device P00 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server P20, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system P24. Inaddition, it may be implemented in a personal computer such as a laptopcomputer P22. Alternatively, components from computing device P00 may becombined with other components in a mobile device (not shown), such asdevice P50. Each of such devices may contain one or more of computingdevice P00, P50, and an entire system may be made up of multiplecomputing devices P00, P50 communicating with each other.

Computing device P50 includes a processor P52, memory P64, aninput/output device such as a light-emitting portion P54, acommunication interface P66, and a transceiver P68, among othercomponents. The device P50 may also be provided with a storage device,such as a microdrive or other device, to provide additional storage.Each of the components P50, P52, P64, P54, P66, and P68, areinterconnected using various buses, and several of the components may bemounted on a common motherboard or in other manners as appropriate.

The processor P52 can execute instructions within the computing deviceP50, including instructions stored in the memory P64. The processor maybe implemented as a chipset of chips that include separate and multipleanalog and digital processors. The processor may provide, for example,for coordination of the other components of the device P50, such ascontrol of user interfaces, applications run by device P50, and wirelesscommunication by device P50.

Processor P52 may communicate with a user through control interface P58and light-emitting portion interface P56 coupled to a light-emittingportion P54. The light-emitting portion P54 may be, for example, a TFTLCD (Thin-Film-Transistor Liquid Crystal Light-emitting portion) or anOLED (Organic Light-emitting Diode) light-emitting portion, or otherappropriate light-emitting portion technology. The light-emittingportion interface P56 may comprise appropriate circuitry for driving thelight-emitting portion P54 to present graphical and other information toa user. The control interface P58 may receive commands from a user andconvert them for submission to the processor P52. In addition, anexternal interface P62 may be provided in communication with processorP52, so as to enable near area communication of device P50 with otherdevices. External interface P62 may provide, for example, for wiredcommunication in some implementations, or for wireless communication inother implementations, and multiple interfaces may also be used.

The memory P64 stores information within the computing device P50. Thememory P64 can be implemented as one or more of a computer-readablemedium or media, a volatile memory unit or units, or a non-volatilememory unit or units. Expansion memory P74 may also be provided andconnected to device P50 through expansion interface P72, which mayinclude, for example, a SIMM (Single Inline Memory Module) cardinterface. Such expansion memory P74 may provide extra storage space fordevice P50, or may also store applications or other information fordevice P50. Specifically, expansion memory P74 may include instructionsto carry out or supplement the processes described above, and mayinclude secure information also. Thus, for example, expansion memory P74may be provide as a security module for device P50, and may beprogrammed with instructions that permit secure use of device P50. Inaddition, secure applications may be provided via the SIMM cards, alongwith additional information, such as placing identifying information onthe SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer or machine-readable medium, such as the memory P64, expansionmemory P74, or memory on processor P5 that may be received, for example,over transceiver P68 or external interface P62.

Device P50 may communicate wirelessly through communication interfaceP66, which may include digital signal processing circuitry wherenecessary. Communication interface P66 may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication may occur, for example, through radio-frequencytransceiver P68. In addition, short-range communication may occur, suchas using a Bluetooth, Wi-Fi, or other such transceiver (not shown). Inaddition, GPS (Global Positioning System) receiver module P70 mayprovide additional navigation- and location-related wireless data todevice P50, which may be used as appropriate by applications running ondevice P50.

Device P50 may also communicate audibly using audio codec P60, which mayreceive spoken information from a user and convert it to usable digitalinformation. Audio codec P60 may likewise generate audible sound for auser, such as through a speaker, e.g., in a handset of device P50. Suchsound may include sound from voice telephone calls, may include recordedsound (e.g., voice messages, music files, etc.) and may also includesound generated by applications operating on device P50.

The computing device P50 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone P80. It may also be implemented as part of a smartphone P82, personal digital assistant, or other similar mobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium”“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a light-emittingportion device (e.g., a CRT (cathode ray tube) or LCD (liquid crystallight-emitting portion) monitor) for light-emitting portioninginformation to the user and a keyboard and a pointing device (e.g., amouse or a trackball) by which the user can provide input to thecomputer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can beany form of sensory feedback (e.g., visual feedback, auditory feedback,or tactile feedback); and input from the user can be received in anyform, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Relative terms, such as lessor, fewer, less than, reduced, greater,higher, increased, etc., can represent that the disclosed examples canbe implemented with fewer cameras than conventional solutions. V-HMDsconstructed in accordance with this disclosure need not provide orspecify a particular decrease in the number of cameras, given bothcameras can realize the same functionality and/or capability. As such,it is not necessary to achieve for a particular and/or specificimprovement(s) by reducing the needed cameras and, thus, need not, andwill not, be specified herein. Further, terms such as, but not limitedto, approximately, substantially, generally, etc. are used herein toindicate that a precise value or range thereof is not required and neednot be specified. As used herein, the terms discussed above will haveready and instant meaning to one of ordinary skill in the art.

Moreover, use of terms such as up, down, top, bottom, side, end, front,back, etc. herein are used with reference to a currently considered orillustrated orientation. If they are considered with respect to anotherorientation, it should be understood that such terms must becorrespondingly modified.

Further, in this specification and the appended claims, the singularforms “a,” “an” and “the” do not exclude the plural reference unless thecontext clearly dictates otherwise. Moreover, conjunctions such as“and,” “or,” and “and/or” are inclusive unless the context clearlydictates otherwise. For example, “A and/or B” includes A alone, B alone,and A with B.

Additionally, connecting lines and connectors shown in the variousfigures presented are intended to represent exemplary functionalrelationships and/or physical or logical couplings between the variouselements. It should be noted that many alternative and/or additionalfunctional relationships, physical connections or logical connectionsmay be present. Moreover, no item or component is essential to thepractice of this disclosure unless the element is specifically describedas “essential” or “critical”. Additionally, the figures and/or drawingsare not drawn to scale, but rather are drawn for clarity of illustrationand description.

Although certain example methods, apparatuses and articles ofmanufacture have been described herein, the scope of coverage of thispatent is not limited thereto. It is to be understood that terminologyemployed herein is for the purpose of describing particular aspects, andis not intended to be limiting. On the contrary, this patent covers allmethods, apparatus and articles of manufacture fairly falling within thescope of the claims of this patent.

What is claimed is:
 1. A method comprising: providing a head-mounteddisplay (HMD) having an imaging sensor, the imaging sensor includingcolor-sensing pixels, and infrared (IR) sensing pixels amongst thecolor-sensing pixels; capturing, using the imaging sensor, an imagehaving a color portion and an IR portion; forming an IR image and acolor image from the captured image by: forming the IR image from atleast some of the IR portion from the captured image; forming the colorimage by replacing the IR portion of the captured image with color datadetermined from the color portion of the captured image and locations ofthe IR-sensing pixels in the imaging sensor; performing a first trackingbased on the IR image; performing a second tracking based on the colorimage; and performing a third tracking based on the IR image and thecolor image.
 2. The method of claim 1, further comprising: performing afourth tracking based on the IR image; and performing a fifth trackingbased on the color image.
 3. The method of claim 1, wherein the secondtracking includes six degrees of freedom tracking and the third trackingincludes hand tracking.
 4. The method of claim 3, wherein the fourthtracking includes IR optical tracking of a controller.
 5. The method ofclaim 1, further comprising providing the color image for display in theHMD.
 6. The method of claim 5, further comprising at least one ofcropping or scaling the color image before the color image is displayedin the HMD.
 7. The method of claim 1, wherein at least two of the image,the color image, and the IR image differ in at least one of a size, adimension, a sampling rate, or a resolution.
 8. The method of claim 1,wherein the IR-sensing pixels being amongst color-sensing pixelscomprises at least one of: the IR-sensing pixels being at leastpartially encircled by at least some of the color-sensing pixels; andthe color-sensing and IR-sensing pixels forming a regular pattern.
 9. Ahead-mounted device (HMD) comprising: an imaging sensor to capture animage using color-sensing pixels and infrared (IR)-sensing pixelslocated amongst the color-sensing pixels, the captured image having acolor portion and an IR portion; a reconstructor configured to remove atleast some of the IR portion from the image, and form an IR image fromthe removed IR portion; a first tracker configured to perform firsttracking using the IR image; an image modifier to form a color image byreplacing the removed IR portion with color pixels determined based onthe color portion of the image and locations of the removed IR-sensingpixels in the image; a second tracker configured to perform a secondtracking based on the color image; and a third tracker configured toperform tracking based on the IR image and the color image.
 10. The HMDof claim 9, wherein IR-sensing pixels replace at least some greensensing pixels of the color-sensing pixels.
 11. The HMD of claim 9,wherein at least two of the image, the color image, and the IR imagediffer in at least one of a size, a dimension, a pixel density, or aresolution.
 12. The HMD of claim 9, wherein the IR-sensing pixels andthe color-sensing pixels of the imaging sensor are arranged in a regularpattern such that each 2×2 block of pixels in the imaging sensorincludes three color-sensing pixels and one IR-sensing pixel.
 13. TheHMD of claim 12, wherein the three color-sensing pixels of each 2×2block of pixels in the imaging sensor include a red-sensing pixel, ablue-sensing pixel, and a green-sensing pixel.
 14. The HMD of claim 9,further comprising a dual-band filter that passes only visible light anda narrow band of IR light disposed inside the HMD such that the imagingsensor is exposed to only the visible light and the narrow band of IRlight that passes through the dual-band filter.
 15. The HMD of claim 9,wherein the second tracking includes a six degrees of freedom tracking.16. The HMD of claim 15, wherein the six degrees of freedom tracking isbased on a chroma sub-sampled image generated from the color image. 17.A system comprising: a first imaging sensor included in a head-mounteddevice (HMD) configured to capture a first image having a color portioncaptured by color-sensing pixels of the first imaging sensor and aninfrared (IR) portion captured by IR-sensing pixels of the first imagingsensor; and a second imaging sensor configured to capture a second imagehaving a color portion captured by color-sensing pixels of the secondimaging sensor and an IR portion captured by IR-sensing pixels of thesecond imaging sensor; at least one memory including instructions; andat least one processor that is operably coupled to the at least onememory and that is arranged and configured to execute instructions that,when executed, cause the system to: form a first IR image using the IRportion of the first image; form a second IR image using the IR portionof the second image; perform an optical tracking based on at least oneof the first IR image or the second IR image; form a first color imageby replacing the IR portion of the first image with color datadetermined based on the color portion of the first image and a locationthe IR portion of the first image; form a second color image byreplacing the IR portion of the second image with color data determinedbased on the color portion of the second image and a location of the IRportion of the second image; perform a six degrees of freedom trackingbased on one or more of the first color image and the second colorimage; and perform a third tracking based on a stereoscopic image formedfrom the first IR image, the second IR image, the first color image, andthe second color image.
 18. The system of claim 17, whereininstructions, when executed, cause the system to additionally provide atleast one of the first color image or the second color image for displayin the HMD.
 19. The system of claim 17, wherein the IR-sensing pixelsand the color-sensing pixels of the first imaging sensor are arranged ina regular pattern such that each 2×2 block of pixels in the firstimaging sensor includes three color-sensing pixels and one IR-sensingpixel.
 20. The system of claim 19, wherein the three color-sensingpixels of each 2×2 block of pixels in the first imaging sensor include ared-sensing pixel, a blue-sensing pixel, and a green-sensing pixel.